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MEMORANDUM 



TO: Members, Legislative Audit Committee 

FROM: Tori Hunthausen 

DATE: October 11,2001 

RE: Assessment of the Department of Revenue's Process Oriented Integrated System 

(POINTS). 



During the June 26, 2001, Audit Committee meeting, a request was made for Audit Division staff to 
compile information and provide the Committee an independent assessment of the status of the Process 
Oriented Integrated System, administered by the Department of Revenue. 

Attached please find the results of our work. We will be discussing this assessment at the October 18, 
2001, meeting. If you have questions please call me at 444-3122. 
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Introduction 

The 1 997 legislature passed HB 1 88, which included authorization of $1 4 million in bonding 
authority for the Department of Revenue (DOR) to develop an integrated revenue and tax system. 
HB 15, passed during the 1999 legislature, provided an additional $18 million in bonding authority 
to complete the project and realign the department's business processes. 

On June 26, 2001 , the Legislative Audit Committee requested Audit Division staff compile 
information on the Department of Revenue's Process Oriented Integrated System (POINTS), and 
provide an independent assessment of the status of the system. Our assessment included the 
following: 

■ Interviewing departmental employees 

- Interviewing non-departmental users of the system, specifically Department of Labor and 
Industry staff 

■ Reviewing contract documents, including the Statement of Work and the System Integration 
Agreement 

■ Reviewing Conceptual and Detailed Design Documents 

■ Attending POINTS system navigation training 

■ Reviewing POINTS defect logs 

■ Obtaining an understanding of the defect remediation process 

■ Reviewing regression test results 

■ Observing business processes 

Expenditures related to the ongoing development of POINTS have occurred during fiscal years 
1 998 to present. As of June 30, 2001 , the department had spent $23,431 ,628 (73%) of the total 
authorized bond proceeds. In addition, the department estimates it has spent in excess of $4.7 
million on direct POINTS development costs from sources other than bonding, primarily for 
personal service costs for department employees dedicated to the project during the past four 
years. 

Background and Influences 



Observation: The department undertook several major efforts during the same time 
frame, adding complexity to the development of POINTS. These efforts impacted the 
work environment of both the department's business and technical staffs. 



Project META: 

In 1997, DOR engaged in a business process re-engineering effort. Project META 
(metamorphosis) is a comprehensive, long-term change program undertaken by the department, 
to transform the way it conducts business both internally and externally. Two primary objectives 
of META are to: 

■ strengthen customer service through redesigned processes and enhanced 
technology; and, 

■ improve the net financial results for the State by collecting what is owed from 
taxpayers more efficiently. 



Business Process Re-engineering: 

DOR conducted organization/workplace redesign to support the re-engineered business 
processes. Previously, DOR's worl^force was organized along tax types. As the department 
realigned its core processes, it moved from a 'tax type' focus to an integrated 'account type' 
concept. This business process re-engineering was considered in the design of POINTS, and 
required staff responsibilities to realign as well. 

Wage Based Tax Consolidation: 

In 1 993, then Governor Racicot initiated a study of unemployment insurance (Ul) and withholding 
(WH) tax, resulting in a task force recommendation to consolidate the functions. The intent was to 
have Montana employers begin using the combined UI/WH returns for the first quarter of 1 999. 
To start this process, the 1 997 legislature consolidated the wage-based tax reporting and 
collection functions of the Department of Labor and Industry (DOLI) and DOR. The Montana 
Simplified Tax and Wage Reporting System (STAWRS) project was also initiated to combine 
business and wage filings to the State and Federal governments, collecting UI/WH information on 
a combined form. 

POINTS: 

Integral to META is POINTS, considered to be the infrastructure that would enable the 
department to realize process-oriented and customer-focused business. 

POINTS is a customized software application. According to department personnel, the 20 stand- 
alone, autonomous computer systems previously used to support business functions could not 
exchange information, were not year-2000 compliant, were poorly documented, and contained 
extensive amounts of redundant data. By using a single, process-oriented computer application 
(POINTS), each customer is registered once, with all associated account type information. The 
design of POINTS is intended to maintain an integrated view of each customer and their 
respective account(s) information. Ultimately, the integrated view would provide efficiencies for: 

■ enhanced customer service, 

■ reduced data redundancy, and 

■ improved compliance capabilities of the department. 

Evolution of POINTS Development: 

POINTS was planned to be developed in two major phases. POINTS I development, which 
began in May 1 998, was to replace eight of the department's existing systems (legacy systems), 
support wage-based account types (UI/WH), and provide the 'foundation' to support the common 
business functions for each of the remaining account types. POINTS II is an extension of POINTS 
I, intended to add property and individual/corporation tax types to the application. 

The request for proposal (RFP) initiated for POINTS I was sent to prospective vendors in 
November 1 997. The following depicts the schedule of events: 

Issue RFP 11/28/97 

Deadline for written questions 12/18/97 

Written Answers distributed 1 2/24/97 

Proposals Due 1/26/98 (extended from 1/5/98) 

Contract Award 3/05/98 

Contractor on site 4/06/98 

The scope of POINTS I work entailed in the RFP included the design, development, and 
implementation of a process-oriented, integrated tax system. The system was to support the 
wage-based taxes, with the intent that other taxes types would be included in future phases 
subject to funding availability. The RFP also included a needs analysis for property valuation. 
The needs analysis would be a starting point for a separate RFP, or negotiation with the POINTS 
I contractor, for developing the property valuation portion in a later phase. 



The contract for POINTS I was signed on March 26, 1 998. The fixed price for the contract was 
$1 0,990,239. The Statement of Worl< (contract) was a cooperative-type agreement where the 
responsibilities and the work were shared between the department and the contractor. 



The POINTS I 'foundation' included the development of five CORE modules: 

■ Registration - Captures basic customer information, including which taxes affect the 
customer. 

■ Forms and Correspondence - Production of standardized department forms and 
correspondence for mass mailing to customers, including experience rating reports, 
quarterly tax statements and statements of account. 

■ Returns Processing - Captures information from the tax return and records it in the 
system according to customer account. 

■ Accounting - posts payments to individual accounts, tracks balances, creates 
refunds and posts payments to the State's revenue accounts. This module 
exchanges information with the Statewide Accounting, Budgeting, and Human 
Resource System (SABHRS). 

■ Case Management - Used extensively for collections and audits, and to track case 
information based on individual accounts. 



Defect Management 



Observation: POINTS is still not operating at design specifications. For the past twenty- 
two months, DOR has been operating in a systems development and production 
environment, identifying mission critical defects and working on resolution, while 
supporting business functions, and performing workarounds. Adding additional account- 
type information and corresponding department business rules will compound the 
complexity and lengthen the timeliness for correcting defects associated with core 
modules. Many department personnel question the long-term stability of the core 
modules. Stabilization of POINTS I is essential to a successful migration of additional 
account types onto the system. 



The Statement of Work, dated September 1 998, states: 

"The teams will make a determination of which defects must be corrected prior to 
implementation and whether the problems can be corrected before the scheduled 
implementation date. In the event the problems cannot be corrected, the earliest 
achievable date will be selected as the new implementation date. Defects not considered 
critical to the operation of the system will be corrected during the six-month warranty 
period." 

POINTS I was placed in production in December 1999. Since implementation, system users 
indicate that anticipated processing efficiencies have not been realized. Rather, throughout its 
twenty-two month life cycle, POINTS has had a number of defects, many of which are critical to 
the mission of the department. 

On December 3, 1 999, the decision was made to place POINTS I into production, while still 
having over 200 identified defects, 91 of which were deemed priority 1 . According to department 
personnel, upon implementation, DOR had 22 days to decide whether to accept the system or 
reject it. They explained that DOR was aware of functionality that was not working upon 
implementation. For example, refund generation and generating direct deposits were not 
working. Although critical to DOR, the management team believed these critical functions could 
be operational in a short time frame and with minimal impact to the business processes. Refund 
generation remains a problem area. 



Defects are referred to as problems identified witli tine system's failure to perform tasks as 
designed. For the first seventeen months, defect remediation equated to crisis management. 
Separation between a system defect, an enhancement, or a training issue, was not always 
apparent, nor was the priority of resolution always considered. Since August 2000, the 
department has been managing defects remediation with both in-house and seventeen 
supplemental contract programmers at the department's estimated cost of $90,000-1 00,000 per 
month. In November 2000, the department acknowledged the extraordinary effort needed in 
relation to system defects, and formed a blitz team. The blitz team's attention turned to critical 
defects. Critical defects were being remedied; however, additional defects not previously 
identified were also being discovered as functionality expanded and the user learning curve 
expanded. In May 2001 , management recognized the need to refine the defect management 
process. The new defect management process improved the logging and prioritizing of identified 
defects. The POINTS action line (PAL) team was established as the first point of user contact, 
rather than contacting the programmers directly. A super-user network relies on process 
expertise to train staff in system use. 

A key addition to the defect correction process is monthly regression testing. Regression 
compares any potential change in programming, with the impact it might have throughout the 
entire system. The goal is for corrections that fail regression testing to not make it to production. 
Previous to regression testing, it was common to repair a defect and place it into production only 
to find that the repair introduced defects in other modules. During the ten-week period from July 
20, 2001 to September 28, 2001 , the department conducted three regression cycle tests. 
Reviewing the department's on-line defect management system, referred to as HEAT, we tracked 
progress in addressing defects throughout the ten-week period. 

As of September 28, 2001 there were 552 reported defects/deficiencies outstanding. The 
following identifies the department's ranking from mission critical to system enhancements. 



Mission Critical: 


205 


Priority 1 




119 


Priority 2 




54 


Priority 3 




43 


Priority 4 




8 


Defects: 


429 


Enhancements: 


123 


Total Ref 


)orted 


552 



Based on the defect descriptions on HEAT, it is apparent these defects are impacting the ability 
of the system to perform accurate or normal processing. The causes of the defects can be in a 
number of related areas, including: 

■ program coding is not allowing for accurate processing of data; 

■ department business rules have changed and the code has not been updated; and, 

■ not all business processes are included in the code. 

During the ten-week period, some defects were fixed and closed; however, there were also a 
number of new defects reported as shown below. These new defects are included in the reported 
total of 552. 

Testing Status 
July 20, 2001 to September 28, 2001 

Defect Ranking Fixed Reported (New) 



Mission Critical: 


31 


52 


Priority 1 






20 


20 


Priority 2 






6 


7 


Priority 3 






2 


5 


Priority 4 









2 


Enhanceme 


nts: 


10 


22 



Total 69 108 



Many of the POINTS I defects may also have an impact on POINTS II. The department's POINTS 
II team has identified a significant number of defects that may impact the critical path to 
implementation of POINTS II. The majority of these are in the Accounting module. 

We interviewed staff on the subject of the defect management process, focusing on user 
involvement in and understanding of defects processing and overall management. The 
interviews indicated 29 of 33 users are aware of the process and points-of-contact. Users noted 
defects ranking does not always relate directly to the workload priorities. As POINTS defects 
develop, staff indicates that the capability to perform work is limited and in some cases halted 
until the defect is fixed. 

Progress towards defect remediation is difficult to quantify. Although the department has 
implemented a process to prioritize defects and identify those considered mission critical, there 
are system design impairments effecting the timeliness of corrective action. Personnel describe 
the structure of the programming for POINTS as extensive and difficult to follow. For example, 
programming sub-routines are not easily identifiable. The system is designed with programming 
'packages' within each core module. Personnel explained the enormity of intelligence coded 
within a given package. Only one programmer can work in a package at a time. This is a control 
established to protect the integrity of the fixes being made, without another programmer over- 
writing code. Therefore, when identified defects co-exist in the same programming package, 
multiple programmers are prohibited from working simultaneously within the same package. 
These limitations extend the time needed for efficient diagnosis of problems and changes. 
Maintenance teams expressed the need for several months of uninterrupted focus to correct 
POINTS I defects. 

Data Conversion 



Observation: The data on POINTS has errors introduced at the time of conversion and 
compounded by system defects. Accurate and complete data is essential to the stability 
of POINTS. The Data Focus Group's efforts to correct data errors require technical and 
business staff involvement, shifting some employee resources to data errors. Data 
Focus Group participants are also working on defect resolution. 



Whenever new systems are implemented, it is necessary to convert old files to the new format, 
often a database, so the new system can be put into use. DOR and the contractor shared 
responsibilities for developing programs to extract data from the legacy systems and create files 
to load to the new POINTS database. Starting in September 1 999, the intention was to conduct a 
full conversion load test. After three failed attempts and performance problems, only a partial 
load was achieved in the test environment. In October 1 999, the first full conversion load was 
conducted for beta implementation (production); however, problems resulted and a second 
attempt was required. Staff explained there were numerous problems with the data extracted 
from the Ul and Withholding systems. In some cases a mini-program was written to clean the 
data. Other problems required either a partial or complete reload of data. In November 1 999, 
two more data loads were attempted. The second full load was completed; however, staff 
indicated DOR was continuing to identify conversion data problems. At November-end, 1 999, 
DOR evaluated the known data problems, and staff explained the errors were not critical enough 
to stop implementation, considering year 2000 compliance and the department's business cycle. 

Staff indicated data problems have plagued the POINTS I production system since it was 
implemented. The data problems are described as: inaccurate and incomplete data converted 
from legacy systems; defects in the POINTS I application; and DOR data scripts and program 
fixes intended to correct other problems while actually introducing new data problems. Examples 
of conversion problems indicated on the HEAT system include: 



■ a year 2000 glitch within the delinquent accounts receivable system that was not 
fixed prior to conversion; therefore, converted receivable amounts for delinquencies 
were overstated; 

■ a developer application upgrade was possibly migrated incorrectly, causing loss of 
logic code due to an out-of-space table condition; 

■ a conversion program needs to be changed to correctly store line item type data; 

■ the possibility of the wrong data source for particular extractions; and, 

■ different customers were assigned the same customer identification numbers. 

In September 2001 , a department Data Focus Group was formed comprised of DOR staff. The 
mission of the group is to focus on data integrity issues. We attended the first two meetings of 
the Group on September 1 and October 1 , 2001 . The consensus of the Group is that data 
integrity is a high priority since good data is necessary in POINTS I in order to move towards 
POINTS II. Members of the Group are making progress quantifying the extent of the data 
problems; however, some of the data goes back to 1 998 leaving no good starting point to make 
comparisons. Apparently, there was bad data on the legacy systems; however, there was also 
the ability for staff to manually intervene and adjust data. System generated adjustments are 
designed in POINTS, limiting a user's ability to fix data 'on the fly.' The Data Focus Group is 
looking at the cause and effect of each data problem type to get an idea of how many errors 
exist and to determine how to correct the data along with the underlying defects producing data 
errors. 

Training/Performance 



Observation: In general, staff indicate customer service seems to be less effective with 
POINTS because users either cannot find information due to defects, or the system 
information is too inaccurate to be useful for timely response. DOR needs to produce not 
just a working system, but surround it with effective user procedures and trained and 
receptive users. 



To determine whether POINTS is being used to perform daily functions, we surveyed the users, 
both department and non-department personnel, to determine whether the system is being used 
to conduct business or whether users are performing workarounds to conduct daily business. We 
randomly selected 60 interviewees from a list of 450 POINTS users and contacted and 
interviewed 51 . We considered position responsibilities, and based on how staff described their 
use of POINTS, categorized as follows: 

User Category # Interviewed % of Sample 

■ Have Not Used 18 35% 

■ Minor Users 15 30% 

■ Major Users 18 35% 

We selected our sample from a department provided list of users. As noted above, 35% of our 
sample indicated they do not use the system; however, they offered their opinion based on the 
information provided them along with the users. It is possible these individuals need additional 
training to promote further use of the system, or truly do not have use for the system in their daily 
job. The department should reevaluate its POINTS Users list and determine whether the 
individuals listed require access to perform their job, and if so provide appropriate training. 

Based on the results of our interviews, many users indicate they have not realized process 
efficiencies, effective compliance enforcement, or technology advancements that were promised. 
Users indicated that system defects in design and capabilities are not totally responsible for the 
current status. Department reorganization and realignment to a team concept created staff 
uncertainties regarding daily responsibilities. At the same time, reassignment of Ul activities to 
DOR added to workload confusion. 



We rated minor and major user comments regarding effectiveness of POINTS as poor, adequate, 
or good. Generally, interviewees were comparing POINTS to the legacy systems, defining a 
poor rating as: 

■ Not as easy to use (user friendly) 

■ Takes more time because there are more screens/ menus/ or 

■ The data used is less accurate and requires verification/update. 

Effectiveness received 19 poor, 10 adequate, and 4 good ratings. The primary concerns related 
to the reduction of customer service. Some example problems include: 

■ Employers are not advised of correct unemployment insurance rates, 

■ Tax collections are more difficult to pursue, 

■ Refund checks must be reviewed manually to avoid duplication, 

■ Statements of account (tax bills) are not correct, 

■ Audit assessments cannot be verified, and 

■ Duplicate identification numbers delay customer service. 

Users indicated the department's approach to training was not effective. It was provided when 
the modules were not functioning, was generic in content, and was not timely. Similarly, the 
approach used to prioritize system defects has not adequately considered workload having the 
most direct impact on staff and customers. Consequently, staff "buy-in" to POINTS 
implementation and use was not adequate. 

POINTS I Processing 



Observation: System functionality does not work as designed creating backlog and 
increased workload beyond ordinary business. 



As discussed earlier, in addition to building the core modules for the foundation, POINTS I is 
intended to support unemployment insurance and withholding taxes. We asked management to 
evaluate the department's workload that is not being accomplished and backlogs that are 
accumulating due to the implementation of POINTS. According to management, wage-based 
taxes and related work are most severely impacted, requiring extensive manual intervention to 
process these accounts. 

Management's estimate of staff time needed to resolve 1 8 separately identified backlogs 
impacting wage-based accounts range from 20 to 18,000 hours each, once correlating defects 
are repaired. Management indicated, however, that as a customer's account is reviewed, 
separate backlog resolutions may overlap and will be resolved simultaneously. 

Although individual and corporation tax accounts are not included in the scope of POINTS I, the 
inability to issue accurate statement of accounts and timely refunds also impact the department's 
backlog associated with those account types. 

Management indicates that as part of its systematic plan to review and revise its internal control 
procedures, it is developing a work plan to resolve these backlogs and stay current with day-to- 
day work. 



POINTS II 



Observation: Defects that are currently being worked in POINTS I, may or may not 
impact the account types developing in POINTS II; however, POINTS II may introduce 
additional defects upon the foundation that could effect all account types. There are tax 
specific defects, and common processing defects. Introduction of additional account 
types to the foundation would have a major effect on resources and the ability to stabilize 
POINTS I. 



On July 26, 1 999, an addendum to the original contract was signed for the contractor to develop 
requirement definitions for Property Tax and Income/Corporation Tax. This expansion of the 
original project is referred to as POINTS II. Addendum I included a fixed price of $1 ,546,380. 
The target date for the requirement definitions for Income and Corporate Tax was December 31 , 
1 999 and January 31 , 2000 for Property Tax. Addendum II was signed June 30, 2000, to develop 
software for the Property Tax and Income/Corporation tax applications for a total cost of $1 
million. 

As explained earlier, POINTS I is to establish underlying core modules to support varying account 
types. Business rules common to account types are coded within the modules. Tax-specific rules 
may be add-ons to a core module. As account types are added to the foundation, functionality 
(business rules) specific to the tax type is coded within the core modules. Department personnel 
explained that the POINTS I database is actually replicated in its entirety (copied defects and all) 
for development of POINTS II. As defects are rectified in POINTS I, they are reconciled to the 
development database for POINTS II. Development will always occur in a mirrored environment; 
therefore, personnel explained that when POINTS II is placed in production, POINTS I will no 
longer exist. 

Concluding Observations: 

Continuing to work in the POINTS environment (with workarounds and fixes in major 
defects) is necessary. POINTS I is supporting the wage-based taxes, all customer 
registration, and all revenue accounting. Returning to the legacy systems is not practical, 
given the re-engineered business processes and the department's reorganization efforts. 
Since the system has been used to conduct business for the past several months, the 
possibility of re-converting data is not likely. Prior to conversion, customer totals were 
tracked in different computer systems depending on the tax type. POINTS will post all tax 
revenue types to one customer account 

On an annual basis, the department is faced with an intricacy of deadlines to conduct its 
business. When reviewing the department's 2001 Key Date Calendar, there are 1 12 
days throughout the year listing a deadline to meet including reporting requirements, 
revenue distributions, and processing of returns and payments. Due to the nature of 
DOR'S business cycle, the complex relationship of tax accounting rules, business 
practices and federal government regulations for which department has no control, it is 
difficult for the department to operate in this systems development environment. 

Inaccurate and incomplete data must be fixed. Both incorrect data and defects are 
causing transfers of money, causing time-consuming manual account reviews. Even 
after manual review, users indicate system-generated adjustments sometimes move the 
money again. The erroneous penalty and interest calculation is resulting in inaccurate 
statement of accounts, which cannot be mailed to customers without individual review. 
These defects must be rectified to provide users the ability to conduct their work. 

The department cannot provide a timeframe for obtaining stability in the core modules. 
This information is necessary to move forward in establishing the future operating 
capability of the system. As part of its maintenance plan, the department recently hired a 
consultant to provide an estimate of when and how POINTS I will become stable. 



A reassessment of system capabilities could determine which remain unfulfilled, and 
those that may never be implemented. Although defects have been prioritized from 
mission critical to enhancements, analyzing the business process flow and separating 
system capabilities into "must do", "should do", and "could do", makes good sense. 
Despite the promises of system functionality, and the overwhelming demands currently 
placed on DOR staff, the current system is not stable enough to handle the increased 
complexity of new tax types and business rules. 



Summary 

In the past five months, the department has acknowledged the need to refine its defect 
management process, has established a Data Focus Group to concentrate on data integrity 
issues, and has introduced monthly regression testing to mitigate new defects resulting from 
defect repairs. Critical decision points related to assurances that DOR business processes will 
function accurately are needed. Stability, accuracy, and completeness in core modules are 
needed in order for there to be positive movement toward the department's original goals of: 

■ enhanced customer service, 

■ reduced data redundancy, and 

■ improved compliance capabilities of the department. 
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DATE: October 18, 2001 

TO: Chairman Tester and Members of the 

Legislative Audit- Commjttee 

FROM: ^urtG, Aime, Director^^— 

RE: Department Response to Auditors' Evaiuation of POilMTS 



Thank you For this opportunity to respond to the Legislative Audit Divtsion's assessment 
of the department's Process Oriented integrated Systenns (POiNTS) Project. On 
balance we believe ttie aBsessment presents a fair and balanced review of the project 
to date. We aiso appreciate your auditors' professionalism in tlie course of their review 
of a wide range of project detaifs; they were able to do so with mtnimal disruption. 

In this response, we will offer our comments on the observations and concluding 
observations in the order they appear In the Special Project Report, dated October 11 
2001, 

1. The department undertook several major efforts during the same trmeframe^ 
adding complexSty to the development of POINTS. 

Response: The history of the department's reorganization and concurrent efforts to 
replace our computer information systems is accurately described on pages 1 
through 3 of the report, For purposes of context, these two undertakings, the 
reorganization and the broad-scale replacement of the many department legacy 
computer systems, represent the two largest challenges the department has 
undertakan since the evolution of the department from the old State Board of 
Equalization structure in 1973. Both of these chaltenges, while disruptive in the 
short term, had to be undertaken jointly to achieve a long term goa! of improved 
customer service, operating efficiencies and a more effective approach to 
compliance with Montana's tax statutes, 

2. POINTS Is still not operating at design specifications. For the past 22 months 
the department has been operating in a systems development and production 
environment . . . Stabilization of POINTS I is essential to a successful 
migration of additional account types onto the system. 



Judy M'<^i\2., Governor 

Kurt G. AJme, Director A Legal Affairs i' Dispute Resolution 
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Response; We agree that it has been difficuit to operate in both environments - 
system development and daily production, This has been mside especiaiiy 
chaiienging given the demands on our staff to support both responsibilities in an 
environment where the system is not operating as designed, This results in 
extensive manual intef\'ention into what wouid otherwise be ajtomated processes. 
The report accurately describes the evolution and refinements in the defect 
remediation process and the iterations of our efforts to resolve the large numbers of 
serious defects in the application. Briefly, these iterations inciuced a request of the 
contractor to extend the original six month warranty provided under contract 
(extended an additionai three months at no addittonai cost), suppiementing our IT 
staff with contract programmers, a concentrated "blitz" effort in the late fali of 2000 
continuing to earfy 2001 , to our most recent efforts to refine the process of stabilizing 
the application. In May of this year we made a number of additional refinements to 
the process including: 

■ Realigning management by designating a senior department manager 
responsible for coordinating the POINTS production and development efforts. 

• Refinmg the defect reporting, classification,, prioritization, and tracking 
processes, including a complete review and reprioritfzation of all outstanding 
defects to address the most critical department and customer needs. 

• Establishing a network of "super users" throughout the department as a 
frontline contact/testing/training networl<. 

• improving regular communications on the status of current functionality, 
defects, and progress on repaired defects focused directly to the v/ork units 
impact ed- 

• Refreshing training materials and currlculun by providing both ad hoc training 
as required at the process level and more global orientation to aN of the 

- modules in POINTS. This training has been provided to a wide range of 
department staff from super users to senior management and external users. 
These courses are available to ail users and will be supplemented with role 
specific training materials m the future. 

• Implementing a monthly cycle of regression testing to confirm defect repairs 
work in a simulated production environment before they are migrated into 
daily production. In this manner, we mitigate the potential of introducing new 
defects, 

• Finally, to ensure we optimize our efforts in this area we have recently hired 
an expert in software engineering, Dr, Joel Henry. We have asked him to 
critically review all aspects of our present processes, assist us in quantifying 
and outlining the best approach to achieving stabilization, and help us 
develop a long term maintenance plan. 

We will continue to aggressively pursue POINTS stabilization until it is achieved. 



3. The cJats on POINTS has errors introduced at the time of conversfon and 
compounded by system defects. 

Response: The report accurately describes the problem and the categories: 
converted data, bad data created as a result of def9Cts, and bad data introduced by 
Ecnpts run attempting to clean up otiner data fields. This is a serious problem and 
we are currently developing plans to attack the issue. This week, two teams of 
programmers and key business users are continuing to analyze in more cfetaii the 
te<;hnieal issues surrounding thas problem. Out of this effort we will develop work 
plans for teams of developers and business users to rectify the data issues in a 
manner Ihat best fits into overall project maintenance. 

4. In general, staff indicate customer service seems to be less effective with 
POINTS because users either cannot find information due to defects, or the 
system information is too inaccurate to be useful for timely response. DOR 
needs to produce not just a working system, but surround It with effective 
user procedures and trained receptive users. 

Response: We agree that these are precisely the things we need to do and we are 

working diligently on all three fronts; 

1. Stabilization - as previously discussed under defect remediation; 

2. User procedures - our Change Management Team is working with the 
processes to develop detailed process work flows, monitoring reports and 
user procedures for all major tax types and liquor functions within the 
department; and 

3. Training - afso as previously discussed, on the job training is being 
provided by super users located on each team, global training is currently 
being provide and role-specific training is being developed. 

5. System functionality does not work as designed, creating backlogs and 
increased workload beyond ordinary business. 

Response; We agree that the defects in the system come with a consequence to 
both our staff and our customers. This is a very stressful situation and we wori< 
every day to mitigate adverse impacts on taxpayers and still meet our many 
deadlines. The greatest impact has been our need to manually inteoj'ene in the 
production of statements of accounts (bills] and refunds rn the wage-based tax 
areas. We have provided the auditors a detailed estimate of these backlogs as of 
last month. Between then and now we have repaired a significant defect that is now 
allowing us to send our statements of account. We are also developing work plans 
in the impacted teams which will address both the daily demands and the backlogs. 



Defects that are currently being worked in POINTS I may or may not impact the 
account types developing [n POINTS II; however, POINTS II may introduce 
additional defects . . . Introduction of additional account types to the 
foundation would have a major effect on resources and the ability to stabilize 

POiMTS I. 

Response: Development must be distinguished from implementation, In 
development, we are insisting on thorough testing of the POINTS 11 application 
according to the agreed upon testing plan. We have asked our software engineering 
expert, Dr, Henry, to evaluate the testing done to date and to be done. We have 
shisred his evaluation and recommendations regarding the testing done to date with 
our contractor and are awaiting its reply. In addition, we are contemplating a parallel 
production run as a final, extensive test of the application before implementation. 

Implementation is the second component. We will implement POINTS II according 
to a detailed implementation plan only after we have been convinced by positive test 
results that the system is ready, In addition, final implementation will be coordinated 
with our business cycles to minimize disruption. 

Concluding Observations: Continuing to work in the POINTS environment 
(with workarounds and fixes in major defects) is necessary . . , , Returning to 
the legacy systems Is not practical, given the re-engineered business 
processes and the department's reorganization efforts. Since the system has 
been used to conduct business for the past several months, the possibility of 
re-converting data is not likely. 

On an annual basis, the department is faced with an intricacy of deadlines to 
conduct its business , . . . [I]t is difficult for the department to operate in this 
systems deveiopment Gnvironment 

The department cannot provide a timeframe for obtaining stabiiity in core 
modules. Thts information is necessary to move forward .... As part of its 
maintenance plan, the department recently hired a consultant to provide an 
estimate of when and how POINTS I will become stable. 

A reassessment of system capabilities could determine which remain unfilled 
and those that may never be implemented. Although defects have been 
prioritized from mission critical to enhancements, analyzing the business 
process flow and separating system capabilities Into ''must do," ''should do," 
and 'could do" makes good sense . . . , [Tjhe current system is not stable 
enough to handle the increased complexity of new tax types and business 
rules. 

Response: These concluding observations make a number of very significant 
conciusions with which we agree: 



• Our assessment is also that the best solution to th^ present situation is to 
proceed to stabilize the application and not return to the legacy systenns. In 
addition la the reasons set forth in the report, returning to the legacy systems 
is not advised because: 

o Technically - reconverting over 22 months of transaction data 

backward plus corrections and adjustments would be very expensive in 

both time and dollars - meanwhile, we would have to maintain and 

operate POINTS; 
o Interfaces with major systems such as MISTICS and SABHRS would 

require extensive rework; • . 

o Customer accounts and cases are not bacl<wards compatible; 
c Return to the old COGS systems would not support the current law 

changes; 
G Peripherals, would also have to be modified backv/ard, such as 

scanner, magnetic tapes, ACH debits and credits; 
o Staffing - programming staff qualified and trained in the legacy 

computer languages would be a sehous challenge; 
o Costs, including costs to retrofit legacy - in the milfions, and then 

ultimately a replacement scenario somewhere downstream - also in 

the millions. 

• It is difficult to operate in this environment and we are open to constructive 
criticism white we actively seek every opportunity to improve our management 
and maintenance effort lo work through these difficult issues. We must do so 
in the most efficient and cost-effective manner within our means, 

• We cannot at this time provide an estimate of when we will be through the 
most chtical defects. However, we have hired an individual with expertise in 
this area to critically review and recommend improvements to allow us to both 
improve our maintenance effort and quantify when and how v/e can obtain 
stability, 

• We will consider all suggestions to improve our defect prioritization efforts. 
We do want to clarify that our present system does attempt to prioritize defect 
fixes in accordance with their importance to business functions with input from 
all parties, users, management, technical staff, and external sources. 

Finally, I want to assure you that we fully understand the importance of getting 
through these difficult times and putting this disruption behind us. Montanans have 
a right to expect good service from their government and we are fervently striving to 
get there. 



